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with high resolution data acquisition device as compared to other biological 
signals. Changes in the human brain’s electrical activity occur very quickly, 

Keywords: thus a high resolution device is required to determine the emotion precisely. 
In this study, we will prove the strength and reliability of EEG signals as an 
emotion recognition mechanism for four different emotions which are happy, 

Electroencep halogram sad, fear and calm. Data of six different subjects were collected by using 

emotion BrainMarker EXG device which consist of 19 channels. The pre-processing 
stage was performed using second order of low pass Butterworth filter to 
remove the unwanted signals. Then, two ranges of frequency bands were 
extracted from the signals which are alpha and beta. Finally, these samples 
will be classified using MLP Neural Network. Classification accuracy up to 
91% is achieved and the average percentage of accuracy for calm, 
fear, happy and sad are 83.5%, 87.3%, 85.83% and 87.6% respectively. 
Thus, a proof of concept, this study has been capable of proposing a system 
of recognizing four states of emotion which are happy, sad, fear and calm by 
using EKG signal. 
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1, INTRODUCTION 

In psychology, emotion is a complex state of feelings that results in physical and psychological 
changes that influence thought and actions of an individual. Emotion can be considered as an experience of 
an individual due to interactions with the environment. Every day, human produce varieties of emotions and 
it can be related to many possibilities such as events, people, objects, self-appraisal and many more. 
Emotions play an essential role in human’s life and it 1s not consciously controlled. Some of the emotion can 
be easily expressed by facial expressions, speech, behavior and gesture but some are not [1, 2]. Controlling of 
emotions are important especially during decision making, communication and survival. For example, 
people tend to be lose concentration if they are too immersed in their emotion while driving on the road. 
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Some of the possibilities that can affect emotions during driving are stress in a workplace, receiving upsetting 
news, involve in arguments or running late to a certain event. 

Failure to express emotions also may lead a person to emotional upheaval. It can be defined as surge 
of emotion that causes major psychological distress for an individual. It is more likely to happen due to 
unexpected lifestyle changes such as divorce or nervous breakdown which can cause an individual to 
experience trauma and mental illness. There are a lot of factors that can cause an individual to have 
emotional upheaval such as mental illness like bipolar disorder, depression and anxiety. People with mental 
illness are more likely to have emotional upheaval of unpleasant emotion [3]. 

Thus, it is proven that identifying emotions of an individual in most of the situation is vital. A lot of 
researches have been performed in order to recognize emotions from speech, facial expressions or both [4-6]. 
However, these methods have limitations and challenges as not all emotions can be obviously expressed 
through facial expression, speech and actions. Outward expression can be controlled due to environment or 
some other circumstances [7-9]. 

Recently, biological signals such as photoplethysmogram (PPG), electroencephalogram (EEG), 
electrocardiogram (ECG) and acceleration plethysmogram (APG) have been proposed as the method to 
detect emotion variabilities. All this information can be obtained through physiological instruments that 
measure the heart rate, blood pressure, oxygen saturation levels, blood glucose, nerve conduction and brain 
activity. However, this study will be focus more on recognizing emotions by using EKG signals. This is 
because it can detect human brain activity accurately with high resolution data acquisition device as 
compared to other biological signals [10-12]. Changes in the human brain’s electrical activity occur very 
quickly, thus a high-resolution device is required to determine the emotion precisely [13, 14]. EEG signals 
are measured through electrical brain waves by placing the scalp over multiple areas of the brain as shown in 
Figure 1. 





Figure 1. Recording user’s brain wave using EEG [15] 


There are many advantages of using EEG signals and one of them is that emotions cannot be 
freely controlled which makes it more accurate as compared to other methods such as facial expressions, 
speech etc. Furthermore, EEG signal is continuously available at all time, even though the person is asleep or 
awake [16]. Hence, this study proposed a system of recognizing four states of emotion which are calm, fear, 
happy and sad using EEG signal. The objectives of this study are; to study the EEG signals and its 
characteristics, to develop the EEG based recognition system as well as to assess the performance of the 
system. 

The rest of this paper is organized as follows. Section 2 summarizes the previous works that are 
related to this study while Section 3 presents the research methodology by explaining the data acquisition, 
preprocessing, future extraction and classification of emotions processes. Further discussion and results will 
be further elaborated in Section 4. Finally, Section 5 concludes the study as well as mapped the future work. 


2. RELATED WORKS 

In recent years, there has been a growing interest from all researchers around the world in 
recognizing variability of human’s emotion from facial expressions, speech and both. However, in this study, 
it is more focused on emotion recognition that uses electroencephalogram (EEG) signals as it 1s relatively a 
new method used by researchers. There are a lot of techniques that have been used in analyzing the EEG 
signals as well as to classify those signals into various types of emotions. 

Horlings et al. in [17] access human emotion by using EEG signals. Two states that have been put 
into attention are valence and arousal. Some pictures are showed to the subjects in order to arouse emotions. 
It is found out that Support Vector Machine (SVM) performs the best in emotion classification where 32% 
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and 37% of classification rates for valence and arousal are achieved respectively. In addition, 
higher performance of classification rates was obtained by using only the extreme values where the rates for 
valence are 71% and arousal are 81%. However, this system did not consider the variety among different 
people and conditions as new samples that were applied and gave only 35% correct accuracy rate. 

Nie et al. in [18] classified two types of emotions which are positive and negative, by using EEG 
signals. Bandpass filter was applied in order to extract the EEG data to delta, theta, alpha, beta and gamma 
waves. Based on the result, alpha, beta and gamma bands showed higher performance in detecting emotions 
compared to delta and theta bands. This can be concluded that bands with higher frequency are essential in 
recognizing emotions. However, 87.53% of average accuracy rate was obtained which provided room of 
improvement as 12.47% error exists. 

Soleymani et al. in [19] proposed a different approach in recognizing emotion by using EEG signals 
and facial expressions. The study used frontal view video in order to capture the facial expressions for each 
subject. Subjects were asked to watch 20 short videos from famous commercially produced movies to evoke 
different emotions. Unwanted artifacts and noise are filtered by preprocessing and a sampling rate of 256Hz 
were used in this study. Long Short Term Memory Neural Networks technology was applied and in order to 
segment and classify data, Conditional Random Fields (CRF) was employed instead of Hidden Markov 
Models (HMM). Results showed electrodes positioned on the frontal, parietal and occipital lobes with high 
frequency obtained higher correlation which carries important information regardless of the contamination 
from facial muscular activities. However, the results will be affected if the subjects are not in stationary 
position which error can easily occur and delay can be experienced due to continuous annotation of facial 
expressions. 

Although there are quite a lot of researches being made to differentiate the variability in human’s 
emotion, including the EEG itself, there are still improvements that can be made in order to increase the 
accuracy of the results obtained. Thus, this study will propose an emotion recognition technique using 
EEG signal. 


3. RESULTS AND ANALYSIS 

Methodology is the analysis of specific procedures or methods used to analyze the information in 
order to obtain the result required. This chapter will briefly discuss about the proposed methodology used to 
study the recognition of emotion using EEG signal. In this study, calm, fear, happy and sad are the four 
emotions to be recognized. Four steps are used to detect and recognize emotion variabilities from individuals 
which are data acquisition, pre-processing, feature extraction and classification as shown in Figure 2. 
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Figure 1. Method used for emotion recognition using EEG signal 





3.1. Data Acquisition 

Data acquisition is the first part of the process where the EEG signals are acquired from six different 
subjects by using an EEG device from Kulliyyah of Information Computer and Technology (KICT), 
named Brainmarker. The device is connected to 19 electrodes which represents the 19 channels of EEG 
signals as shown in Figure 3. 








Figure 2. EEG device namely brainmarker EXG [20] 
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All subjects are students from International Islamic University Malaysia (IUM) with age range 
from 22 to 24 years old and in good health condition. These subjects need to watch four videos in order to 
stimulate calm, sad, fear and happy emotions. Before the videos are started, the subjects were given a 
headphone so that they will be more focused on the video without any external distraction. However, 
EEG signal is very sensitive to movements. Therefore, all subjects are advised not to make any movement 
and minimize eye blinking as it will give major effect to the EEG signal. 


3.2. Pre-processing 

After the EEG signal is acquired, these data signals need to go through pre-processing. 
Pre-processing 1s a state of process to remove noise from the signals. Since brain activity only produces very 
weak signals, EEG signals will contain a lot of background noise [21]. There are many sources of noise such 
as Static electricity or electromagnetic fields produced by the nearby devices, artifacts and any other external 
influences. In this study, second order low pass Butterworth filter is used at the pre-processing stage as all the 
relevant information especially for emotion classification are found in the frequencies below 30Hz. 


3.3. Feature Extraction 

The second stage of this study is feature extraction where all the useful information from the signals 
will be extracted. Based on previous study, high frequency bands play an essential role in recognizing human 
emotion variabilities [18]. Therefore, in this part, EEG signals will be selected based on their frequency band 
which are alpha bands and beta bands. The range of frequencies for the bands are between (8-13)Hz and (13- 
30)Hz respectively. In this study, wavelet transform is used in order to split the signals into selected 
frequency bands. Multi-resolution analysis of “db8” wavelet function is used for decomposing the EEG 
signals into five levels and two frequency bands which are alpha and beta. 


3.4. Classification 

After extracting the selected features, the classification of emotion will be done by suitable 
classifiers where this is the last stage of this study. Classification is the process of grouping the related data 
into a single group. Hence in this study, Multilayer Perceptron (MLP) is used. It is a feedforward artificial 
neural network model that plots sets of input data onto a set of outputs. All the network parameters adapted 
are shown in Table 1. 


Table 1. Parameter of Neural Network 


Parameters Values 
Number of input layer 12 
Number of hidden layer 2 
Number of output layer 2 
Number of epochs 1000 
Mean square error 0.1 


In this study, the result of classification process will be based on the Russell’s Circumplex Model of 
Effect as it is one of the commonly used model of emotions as shown in Figure 4 [22-25]. Each emotion is 
placed according to their level of arousal and valence. Arousal ranges from excited to calm while valence 
ranges from pleasant to unpleasant. 
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Figure 3. Russell's circumplex model of effect [24] 
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4. RESULTS AND ANALYSIS 

In this study, EEG data were taken from six different subjects age ranged from 22 to 24 years old 
and all of them were in good health condition. They are required to watch four different videos to stimulate 
calm, fear, happy and sad emotions. In order to acquire the EEG data, subjects were required to wear a cap 
that consist of 19 electrodes. The cap was connected to a device and a laptop. Brainmarker software are used 
for the data acquisition process. Based on previous research, some of the electrodes gave higher accuracy rate 
in classifying emotions [24]. 

Therefore, in this study, T8, F3, P3 and P4 channels were used to analyze happy, calm, fear and sad 
respectively. The number of samples in each signal is depends on the time duration of the video clips. 
Figure 5 and Figure 6 display the raw data of subjects | and 2 respectively for all emotion state. 
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Figure 4. Raw EEG signal for Subject | Figure 5. Raw EEG signal for Subject 2 


The outcome from the first process of this study which is data acquisition from 6 different subjects 
showed that different people have different waveforms for each emotion. The next stage after data acquisition 
is pre-processing. The noise or unwanted signal may be caused due to the movement of the subjects or the 
instability of the subjects during data acquisition process. Therefore, preprocessing is needed to get the 
smooth signal waveforms. In this study, second order low pass Butterworth filter with normalized cutoff 
frequency of 0.25 Hz is used. The filtered signal of subjects 1 and 2 are shown in Figure 7 and Figure 8. 
The output showed smooth waveforms of EEG signals. 
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Figure 6. Filtered EEG signal for Subject 1 Figure 7. Filtered EEG signal for Subject 2 


After undergoing the pre-processing stage, the signals are then extracted according to frequency 
bands. High frequency band which are alpha and beta, play an important role in emotion classification. 
Hence, signal with frequency (8-13) Hz and (13-30) Hz are extracted to represent alpha and beta respectively 
by using Wavelet Transform. Figures 9 until 16 show the alpha and beta signals of each emotion for subjects 
1 and 2. 
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Figure 9. Alpha and beta signal of subject | 
(Calm) 
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Figure 11. Alpha and beta signal of subject 1 
(Happy) 
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Figure 13. Alpha and beta signal of subject 2 
(Calm) 
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Figure 15. Alpha and beta signal of subject 2 
(Happy) 
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Figure 10. Alpha and beta signal of subject 1 
(Fear) 


crs Rind WAVE OF SUBJECT 1 (SAD) 


ol i Kh Ny 


0 


Arner heda 


hai Ma) ‘i 









Iw Ali i ( MI ju ) Nev lh mn 





4000 Sooo too 
Samnie Numher 
BETA WAVE OF SUBJECT 7 (SAD) 


ionomers 


02 2020 3000 4000 Sooo 7230 
Sample Number 


700 sone S00 toon 


Amplituds 
bh &£ ow 


Q 70o0 $000 soo0 089 10200 


Figure 12. Alpha and beta signal of subject 1 
(Sad) 
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Figure 14. Alpha and beta signal of subject 2 
(Fear) 


ALP HA WAVE OF SUBJECT 2 (SAD) 


| i kal i Wy AMA 


WW wit h Ay i a “fl 


41na) MED 





“ax teal ab eal fan wah Wah 


BETA WAVE OF OF SUBJECT 2 (SAD) 







AY Nae 


10500 


S205 4000 S000 60 eond 


Semple Nuenber 


7o00 8000 SoD 


Figure 16. Alpha and beta signal of subject 2 
(Sad) 
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The last stage of this study is classification. This is the part where all the signals will be classified 
according to the emotion. The data extracted from feature extraction process will be the input for this stage. 
In this study, MLP neural network is used by using toolbox in MATLAB R2016a. The result of the 
classification process for subjects | and 2 are shown in Figure 17 and Figure 18 respectively. 
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Figure 17. Emotion classification for Subject 1 Figure 18. Emotion classification for Subject 2 


Based on the result of classification process in Figure 17 until 18, majority of the signals are in the 
right quadrant of emotion as referring to Russell’s Circumplex Model of Effect. Figures 4 show the scatter 
plot of the classification result after MLP neural network is applied. In each scatter plot, 
four emotions which are happy, fear, calm and sad are highlighted. For happy emotion, it 1s indicated as 
having positive intensity for both valence and arousal while sad emotion located at the quadrant of negative 
for both valence and arousal. For sad emotion, almost the entire plot indicates negative level of both valence 
and arousal. Meanwhile, fear emotion corresponds to positive arousal and negative valence. 

However, based on the classification result, 17% of sad emotion were detected in quadrant of 
positive arousal. Due to psychological reason, when someone experience sad emotion, they tend to have 
anxiety or upset as well. Furthermore, the fear of losing someone also may contribute to high intensity of 
arousal as well. Table 2 shows the classification accuracies of EEG signal according to emotions for all 
subjects. 


Table 1. Classification Accuracies of EEG Signal According to Emotions for All Subjects 


Subjects Calm Fear Happy Sad 
Subject 1 85% 92% 92% 90% 
Subject 2 84% 85% 85% 59% 
Subject 3 85% 81% 89% 89% 
Subject 4 719% 88% 717% 90% 
Subject 5 82% 89% 82% 82% 
Subject 6 86% 89% 90% 86% 


Based on the table above, there are varying classification accuracy result as this is due to multiple 
emotions at the same time which causes the categorization of emotions to be mix up. However, by using the 
proposed method, classification accuracy up to 91% is achieved. The average classification accuracy for 
calm, fear, happy and sad are 83.5%, 87.3%, 85.83% and 87.6% respectively. 


5. CONCLUSION 
This study inspects the capability of EEG signal in recognizing variability of human emotion. 


Four steps are used to detect and recognize emotions which are data acquisition, pre-processing, feature 
extraction and classification of emotions. Chapter Four shows the results obtained by using the four processes 
mentioned before. This study proves that MLP neural network contributes well in classifying human 
emotions. Taking into consideration, the analysis that has been made in previous chapter, this study shows an 
improvised emotion recognition’s system especially in differentiation four different emotions which are 


happy, sad, fear and calm. 
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